Methods and systems tracking head mounted display (HMD) and calibrations for HMD headband adjustments

ABSTRACT

Methods and systems are provided for head mounted display (HMD) implementations. One example implementation includes a circuit for communicating with a computing system that processes multimedia content for display in the HMD. A front unit of the HMD has a screen for displaying multimedia content, and the front unit has a set of LEDs. A rear section of the HMD has a set of LEDs. A connector is provided for coupling the front unit with the rear section of the HMD, such that adjustment of the connector changes a separation distance between at least one of the set of LEDs of the front unit and at least one of the set of LEDs of the rear section. The computing system processes image data captured of the HMD when worn by a user. The image data includes at least one of the set of LEDs of the front unit and at least one of the set of LEDs of the rear section to identify the separation distance for a current adjustment of the connector.

CLAIM OF PRIORITY

The present application is a Continuation of U.S. patent applicationSer. No. 14/658,123, filed on Mar. 13, 2015, (U.S. Pat. No. 9,710,057,issued on Jul. 18, 2017), entitled “Methods and Systems Tracking HeadMounted Display (HMD) and Calibrations for HMD Headband Adjustments”,which is a non-provisional of U.S. Provisional Application No.61/953,730, filed on Mar. 14, 2014, and entitled “Methods and SystemsTracking Head Mounted Display (HMD) and Calibrations for HMD HeadbandAdjustments”, which are herein incorporated by reference.

BACKGROUND 1. Field of the Invention

The present invention relates to methods and systems for trackingposition and orientation of a head mounted display (HMD) andcalibrations made for headband adjustments of the HMD.

2. Description of the Related Art

The video game industry has seen many changes over the years. Ascomputing power has expanded, developers of video games have likewisecreated game software that takes advantage of these increases incomputing power. To this end, video game developers have been codinggames that incorporate sophisticated operations and mathematics toproduce a very realistic game experience.

Example gaming services and systems may include those provided by SonyPlaystation®, which are currently sold as game consoles, portable gamedevices, and provided as services over the cloud. As is well known, thegame console is designed to connect to a monitor (usually a television)and enable user interaction through handheld controllers. The gameconsole is designed with specialized processing hardware, including aCPU, a graphics processor for processing intensive graphics operations,a vector unit for performing geometry transformations, and other gluehardware, firmware, and software. The game console is further designedwith an optical disc tray for receiving game compact discs for localplay through the game console. Online gaming is also possible, where auser can interactively play against or with other users over theInternet. As game complexity continues to intrigue players, game andhardware manufacturers have continued to innovate to enable additionalinteractivity and computer programs.

A growing trend in the computer gaming industry is to develop games thatincrease the interaction between the user and the gaming system. One wayof accomplishing a richer interactive experience is to use wireless gamecontrollers whose movement is tracked by the gaming system in order totrack the player's movements and use these movements as inputs for thegame. Generally speaking, gesture input refers to having an electronicdevice such as a computing system, video game console, smart appliance,etc., react to some gesture made by the player and captured by theelectronic device.

Another way of accomplishing a more immersive interactive experience isto use a head-mounted display. A head-mounted display is worn by theuser and can be configured to present various graphics, such as a viewof a virtual space. The graphics presented on a head-mounted display cancover a large portion or even all of a user's field of view. Hence, ahead-mounted display can provide an immersive experience to the user.Current tracking techniques of an HMD's position and orientation stillneed improvement, to enable improved rendering of content by the HMD.

It is in this context that embodiments of the invention arise.

SUMMARY

Embodiments of the present invention provide methods and systems forenabling head mounted displays (HMDs) to improve tracking of positionand movements of the HMD when a user is wearing the HMD and viewingand/or interacting with multimedia content. The systems and methodenable dynamic calibration to account for adjustments in a headband ofthe HMD, so that distance separations between front tracking LED lightsand rear tracking LED lights can be accurately estimated. Thecalibration can be updated from time to time, such as when a useradjusts the headband size, in between sessions, when sensors determinethat adjustment has occurred, and when other users adjust the headbandfor a better fit. It should be appreciated that the present inventioncan be implemented in numerous ways, such as a process, an apparatus, asystem, a device or a method on a computer readable medium. Severalinventive embodiments of the present invention are described below.

In one embodiment, a method is provided. The method includes capturingvideo frames using a camera. The video frames are configured to capturemarkers on a head mounted display (HMD), and the markers on the HMD areanalyzed in the captured video frames to determine position andorientation of the HMD for processing changes to scenes generated duringrendering of multimedia content that is displayed by the HMD. The methodfurther includes estimating a separation distance between a marker on afront unit of the HMD and a marker on a rear section of the HMD. Thefront unit and the rear section are coupled together by an adjustableheadband. The estimating includes analyzing a plurality of video framesand inertial data captured when the video frames were captured, and theanalyzing produces an estimated separation distance between the markeron the front unit of the HMD and a marker on the rear section of theHMD. The estimated separation distance is used during further trackingof position and orientation of the HMD, as markers on the front unit andthe rear section are captured and analyzed from the captured videoframes.

In another embodiment, a system is provided. The system includes acomputing system and a head mounted display (HMD) in communication withthe computing system. The HMD includes a front unit having a screen fordisplaying multimedia content. The front unit has a marker disposedthereon, and inertial sensors for generating inertial data indicative ofchanges in position and orientation of the HMD. In one embodiment, theinertial data produces a set of values that quantify changes in rotationand changes in position of the HMD. The set of values can be mapped toidentify locations of the markers of the HMD in image data of theplurality of the captured video frames. The HMD also includes a rearsection of the HMD connected to the front unit by an adjustableheadband. The rear section has a marker disposed thereon. The systemfurther includes a camera in communication with the computing system.The camera is configured to capture video frames of the HMD during asession of use. The captured video frames from time to time areconfigured to include the marker of the front unit and the marker of therear section. The computing system processes the captured video framesto identify the markers of the front unit and the rear section againstthe generated inertial data. The processing produces an estimatedseparation between the marker of the front unit and the marker of therear section. The estimated separation is used for improved trackingduring use.

In yet another embodiment, a head mounted display (HMD) is provided. TheHMD includes a circuit for communicating with a computing system thatprocesses multimedia content for display in the HMD. Further included isa front unit of the HMD that has a screen for displaying multimediacontent, and the front unit has a set of LEDs. The HMD includes anaccelerometer and gyroscope disposed in the front unit of the HMD. Arear section of the HMD is provided having a set of LEDs. A headbandconnecting the front unit to the rear section is included, such thatadjustment of the headband changes a separation distance between atleast one of the set of LEDs of the front unit and at least one of theset of LEDs of the rear section. Wherein calibration of the separationdistance is performed from time to time to produce and estimatedseparation distance for tracking of the HMD during use.

Other aspects of the invention will become apparent from the followingdetailed description, taken in conjunction with the accompanyingdrawings, illustrating by way of example the principles of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings.

FIG. 1A illustrates a system for interactive gameplay of a video game,in accordance with an embodiment of the invention.

FIG. 1B illustrates a head-mounted display (HMD), in accordance with anembodiment of the invention.

FIG. 2 illustrates one example of gameplay using the client systemcoupled to the HMD, in accordance with an embodiment of the invention.

FIGS. 3A-3B illustrate an example of a user wearing an HMD where acamera sits at a fixed location, in accordance with an embodiment of theinvention.

FIGS. 3C-1 through 3G-3 illustrate examples of a user wearing an HMDthat is strapped to the user's head via a headband and image datacaptured for the same, in accordance with an embodiment of theinvention.

FIG. 4A illustrates a top view of the HMD, and a camera which ispositioned to be able to identify markers (e.g., LEDs) associated withthe HMD, in accordance with an embodiment of the invention.

FIGS. 4B-4C illustrate examples of a top view of an HMD where aseparation is physically adjusted and associated processing, inaccordance with an embodiment of the invention.

FIG. 4D provides an example flow diagram of operations that can beperformed to identify the separation distance between the front markerson the HMD and the visible rear marker associated with the headband ofthe HMD, in accordance with an embodiment of the invention.

FIG. 4E illustrates another flow diagram example, utilized to makecalibrations to the separation distance between LEDs of the front unitof the HMD and an LED associated with a rear portion of the HMD, inaccordance with an embodiment of the invention.

FIG. 4F illustrates yet another embodiment, wherein examination of videoframes and inertial data is performed to identify separation distancesbetween the front and rear markers of the HMD, in accordance with anembodiment of the invention.

FIG. 5A provides an example of a top view of the HMD, in accordance withone embodiment of the present invention.

FIGS. 5B and 5C illustrate examples and processing for when inertialsensors are added to the rear section (i.e., in addition to just havinginertial sensors in the front unit), in accordance with one embodimentof the present invention.

FIGS. 6A-6C illustrate various views of the HMD, when worn on a head ofa human user, in accordance with one embodiment.

FIG. 7 shows a side view of the HMD, showing that sometimes the headbandcan be pulled and/or twisted, which can trigger a need for recalibrationof the estimated separation distance, in accordance with one embodiment.

FIG. 8 illustrates a user wearing the HMD, during use (e.g., game play),in accordance with one embodiment.

FIG. 9 is a diagram illustrating example components of a head-mounteddisplay 102, in accordance with an embodiment of the invention.

FIG. 10 illustrates components of a head-mounted display, in accordancewith an embodiment of the invention.

DETAILED DESCRIPTION

In one embodiment, the systems and methods described herein provide fortracking the position and movements in position of a head mounteddisplay (HMD) when a user is wearing the HMD and viewing and/orinteracting with multimedia content. The systems and method enablecalibration to account for adjustments in a headband of the HMD, so thatdistances of markers (e.g., LED lights) disposed on the HMD and theheadband are accurately estimated. As described below, the calibrationcan be updated from time to time, such as when a user adjusts theheadband size during a session of use, in between sessions, when sensorsdetermine that adjustment has occurred, and when other users adjust theheadband for a better fit. Methods and systems for performingcalibrations will therefore ensure that the estimated distances betweenmarkers are updated to be as close as possible to the actual currentphysical distances on the HMD, which provides for more accurate trackingof the HMD for position and movement during use. It will be obvious,however, to one skilled in the art, that the present invention may bepracticed without some or all of these specific details. In otherinstances, well known process operations have not been described indetail in order not to unnecessarily obscure the present invention.

In one example, the HMD worn by a user provides the user access to viewrich multimedia content, which can include video games, movies, internetcontent, and other types of interactive and non-interactive content. Thetracking of the HMD is performed using a combination of systems. Thesystems include, without limitation, inertial sensors in the HMD andoptical tracking using one or more cameras. A camera used in opticaltracking can capture video of the user wearing the HMD, so when the usermoves around with the HMD the video frames can be analyzed to determineposition, orientation and movements of the HMD. Broadly speaking, somecontent presented by the HMD is dynamically dependent on movement of theHMD.

For example, if the HMD provides a view into a scene, the user is ableto naturally move his or her head to view other parts of the scene. In avideo gaming example, a user wearing the HMD can move his or her head inany direction to move about in and around a virtual scene. In oneembodiment, the virtual scene is rendered in a rich three dimensional(3D) format. Consequently, in order to smoothly render content in theHMD, the movement of the HMD will be tracked with high fidelity. In oneconfiguration, the HMD is configured to communicate with a client system106, which renders the content presented to the HMD. The content (e.g.,game, movie, video, audio, images, multimedia, etc.), in someembodiments may be streamed from a remote server or servers using cloudgaming infrastructure. In some examples, the content is downloaded tothe client system 106 for rendering and then transferred to the HMD.

As noted above, the tracking may include the use of inertial sensorsthat are disposed within the HMD. Example inertial sensors include oneor more accelerometers and one or more gyroscopes. Some implementationsmay include more or less inertial sensors. In addition to inertialsensors, the HMD can be tracked using a camera. The HMD is, in oneembodiment, configured with several lights (e.g., light emitting diodes(LEDs)), which act as markers. The markers can then be easily identifiedby analyzing, by the client system 106, one or more video framescaptured by the camera. In one configuration, the HMD includes four LEDson the four corners of the front unit 102 a (e.g., also referred toherein as the optics block) and two LEDs on the rear section (e.g., bandadjustment unit 102 b).

The front unit 102 a, in one example, includes a front face and a sideface on each side, wherein the front face and side faces define asubstantially continuous surface. In various examples provided herein,the front LEDs are defined in a housing of the front unit 102 a, anddisposed with transparent plastic that can illuminate when the LEDs areturned on. Further, in some embodiments, the front LEDs are configuredto be partially disposed on the front surface and partially on the sidesurface, to define a partial L-shape, or curved L-shape, or boomerangshape, or a curved rectangle, or curved line, or a spot, or circle, or apattern, or combinations thereof.

This shape allows for tracking of the front unit 102 a when the user isdirectly facing the camera 108 and when the user starts to turn awayfrom direct facing of the camera 108. As the user faces to the side andfurther away from the directly facing the camera, the front LEDs will bevisible until only the LEDs on one side of the front are visible and oneof the LEDs on the backside are visible. This is the transition from thefront LEDs to the front and back LEDs. Due to this transition, as notedabove, the separation distance between the front and back LEDs isneeded, so that accurate tracking can proceed.

Still further, when the user wearing the HMD is facing the camera, thecamera should be able to view all four LEDs. The separation of the fourfront LEDs is known to the client system 106. For example, a geometricmodel of the HMD can be accessed by programs executed on the clientsystem 106, to determine depth (relative to the camera) and orientationof the user's head when wearing the HMD. For instance, because the fourLEDs are, in one embodiment, disposed on the corners (e.g., outlining arectangular shape), it is possible to determine from the captured videoframes if the user is viewing down, up or to the sides.

However, because the interactive content that can be rendered in the HMDcan be virtually boundless, a user is able to view and interact with thevirtual scene in most every dimension. A user wearing an HMD, therefore,may decide to turn his or her head in any direction, which is notnecessarily always forward facing with respect to the camera. In fact,depending on the content rendered (e.g., immersive interactive videogames, moves, virtual tours, clips, audio, and combinations thereof),users will many times be facing to the sides of the camera and directlyaway from the camera.

During such interactive sessions, the camera tracking the HMD will gofrom seeing the front four LEDs to sometimes seeing the side of two ofthe front LEDs and also one of the rear LEDs. Although the front fourLEDs remain in a fixed relative orientation, based on the geometricmodel of the HMD, the rear LEDs may change in position depending on anadjusted setting of a headband of the HMD. For instance, if a user witha smaller head adjusts the headband to fit, the distance between thefront LEDs and the rear LED (e.g., when viewed from the side) will becloser, relative to an adjusted setting for a larger head of anotheruser.

To account for the changes in headband adjustments, a process isconfigured to calibrate the geometric model of the HMD, so that theseparation between the front LEDs and the rear LED (e.g., when viewedfrom the side when the user turns his or her head away from normal tothe camera) can be used to accurately render scene content to the HMDand provide the scenes from the desired perspective, angle and/ororientation. In one implementation, the geometric model is a computermodel, which stores/contains dimensions and/or three-dimensionaloutlines of the HMD 102, similar to what a computer aided design (CAD)drawing may show. However, the geometric model is not displayed as adrawing, but instead is stored as a data set, that is accessible bygames, or movies, or software, or firmware, or hardware, or combinationsthereof to enable accurate tracking.

Still by way of example, the three-dimensional outlines of the HMD 102can include, in one embodiment, outlines of each shape of the HMD andthe shapes of the LED regions, the locations of the LEDs relative toshapes in the outline, the angles and contours of the physicalstructures of the HMD, and data sets that define measurements of thefeatures and constructs of the HMD. In particular, the geometric modelmay include dimensional data that define the exact relative placement ofthe LEDs on the front unit 102 a. However, because the rear LEDs arecoupled to a headband that is adjustable, the separate distance must beupdated during calibration, so that the geometric model can be updatedwith a more accurate distance between the front and rear LEDs.

In one embodiment, the calibration process is configured to initiateafter analysis of the video frames determines that one of the rear LEDsis visible (e.g., starting from when only the front LEDs are visible).For example, at the start of a session (e.g., game play or interactivesession), it is common that the user will face the HMD toward thecamera. At some point, the user will turn his head away from the camera,which will expose at least one of the rear LEDs. At this point, theanalysis of the video frames, which is ongoing, will detect theappearance of the rear LED.

In one embodiment, the process will analyze several frames as the usercontinues to move to associate the visible rear LED and the visiblefront LEDs with inertial data. The inertial data present for each frame,for example, is used to associate an estimated separation distancebetween the visible rear LED and the visible front LEDs. In one example,gyroscope data from the HMD is used to determine the rotation motion bythe user's head, as the HMD moves. Further, by way of example,accelerometer data from the HMD is used to determine movement, such asposition (e.g., tilt/pitch) and rotation.

Thus, using the image data from the captured video frames of the HMD(i.e., when the rear and front LEDs are visible), the inertial data incombination with the image data will render an estimated separationdistance between the front LEDs and the rear LED, for the current sizesetting of the headband of the HMD. This data is then used to calibratethe geometric model of the HMD, which includes the estimated separationdistance. In one embodiment, the calibration can be updated from time totime, and can also be calibrated independently for each side of the HMD.

Once the calibration to the geometric model of the HMD is complete, theuser may proceed to interact during the session. However, once thesession is done, it is possible that a different user may wish access tothe HMD. At such time, it is also likely that the new user will adjustthe headband of the HMD to another size, which will cause a change inthe actual separation distance between the front LEDs and the rear LEDs.In one embodiment, a new session can begin, using either the priorupdated calibration or the dimensions from the original geometric model.

At the start, the game play or interactive session will proceedsmoothly, wherein the scene presented in the HMD will render based onmovements in the user's head. This will be so, while the user wearingthe HMD is facing forward toward the camera, wherein the fixedseparation between the four LEDs in the front of the HMD are known.However, once the user turns away from the camera and the rear LED isfound, the system, without automatic calibration, would see a jump orpop in the content rendered in the HMD. This is so, as the tracking ofthe HMD, which utilizes marker tracking of the LEDs to identifyposition, would be out of sync with the true position of the HMD.

In one embodiment, a determination as to whether re-calibration isneeded is performed each time the user faces the HMD to the side,wherein the front and rear LEDs become visible (i.e., coming from whenonly the front or only the rear LEDs are visible). In one example, ifthe calibration occurred for a current session, and the session ison-going with a current calibration, the system will run a calibrationin the background to determine if the current calibration is stillwithin a pre-defined tolerance margin. For example, if the same useradjusted the headband during game play, or took off the HMD for a minorsize adjustment, or some other person tried on the HMD momentarily, thenthe actual separation between the front and the rear would be differentthan what was used to estimate the separation during an initialcalibration.

The tolerance margin is configured or chosen so that if the newbackground calibration shows that rendering glitches, skips or popswould likely occur (e.g., in the video images rendered in the HMD), thenthe new calibration should become the current calibration.

In still another embodiment, the HMD will include a headband adjustmentdetector that will set a flag. The flag can be read by the system and/orgame executing, which can be used to require recalibration of thegeometric model. For instance, if the user adjusts the headband duringuse of the HMD, the system can be alerted via the flag that thecalibration should be re-run. The same may be true if the adjustmentoccurred because another user tried on the HMD, even if the same sessionis in progress. In still further embodiments, a flag can be generatedupon the start of a new session or when the system detects that the HMDhas been still or has not moved for some time. Such indicators can beviewed as possibility that the headband may have been adjusted, beforethe new session occurred or even during a session.

FIG. 1A illustrates a system for interactive gameplay of a video game,in accordance with an embodiment of the invention. A user 100 is shownwearing a head-mounted display (HMD) 102. The HMD 102 is worn in amanner similar to glasses, goggles, or a helmet, and is configured todisplay a video game or other content to the user 100. The HMD 102 isconfigured to provide an immersive experience to the user by virtue ofits provision of display mechanisms (e.g., optics and display screens)in close proximity to the user's eyes and the format of the contentdelivered to the HMD. In one example, the HMD 102 can provide displayregions to each of the user's eyes which occupy large portions or eventhe entirety of the field of view of the user.

In one embodiment, the HMD 102 can be connected to a computer 106. Theconnection to computer 106 can be wired or wireless. The computer 106can be any general or special purpose computer, including but notlimited to, a gaming console, personal computer, laptop, tabletcomputer, mobile device, cellular phone, tablet, thin client, set-topbox, media streaming device, etc. In some embodiments, the HMD 102 canconnect directly to the internet, which may allow for cloud gamingwithout the need for a separate local computer. In one embodiment, thecomputer 106 can be configured to execute a video game (and otherdigital content), and output the video and audio from the video game forrendering by the HMD 102. The computer 106 is also referred to herein asa client system 106 a, which in one example is a video game console.

The computer may, in some embodiments, be a local or remote computer,and the computer may run emulation software. In a cloud gamingembodiment, the computer is remote and may be represented by a pluralityof computing services that may be virtualized in data centers, whereingame systems/logic can be virtualized and distributed to user over anetwork.

The user 100 may operate a controller 104 to provide input for the videogame. In one example, a camera 108 can be configured to capture image ofthe interactive environment in which the user 100 is located. Thesecaptured images can be analyzed to determine the location and movementsof the user 100, the HMD 102, and the controller 104. In one embodiment,the controller 104 includes a light (or lights) which can be tracked todetermine its location and orientation. Additionally, as described infurther detail below, the HMD 102 may include one or more lights whichcan be tracked as markers to determine the location and orientation ofthe HMD 102 in substantial real-time during game play.

The camera 108 can include one or more microphones to capture sound fromthe interactive environment. Sound captured by a microphone array may beprocessed to identify the location of a sound source. Sound from anidentified location can be selectively utilized or processed to theexclusion of other sounds not from the identified location. Furthermore,the camera 108 can be defined to include multiple image capture devices(e.g. stereoscopic pair of cameras), an IR camera, a depth camera, andcombinations thereof.

In some embodiments, computer 106 can execute games locally on theprocessing hardware of the computer 106. The games or content can beobtained in any form, such as physical media form (e.g., digital discs,tapes, cards, thumb drives, solid state chips or cards, etc.) or by wayof download from the Internet, via network 110. In another embodiment,the computer 106 functions as a client in communication over a networkwith a cloud gaming provider 112. The cloud gaming provider 112 maymaintain and execute the video game being played by the user 100. Thecomputer 106 transmits inputs from the HMD 102, the controller 104 andthe camera 108, to the cloud gaming provider, which processes the inputsto affect the game state of the executing video game. The output fromthe executing video game, such as video data, audio data, and hapticfeedback data, is transmitted to the computer 106. The computer 106 mayfurther process the data before transmission or may directly transmitthe data to the relevant devices. For example, video and audio streamsare provided to the HMD 102, whereas a vibration feedback command isprovided to the controller 104.

In one embodiment, the HMD 102, controller 104, and camera 108, maythemselves be networked devices that connect to the network 110 tocommunicate with the cloud gaming provider 112. For example, thecomputer 106 may be a local network device, such as a router, that doesnot otherwise perform video game processing, but facilitates passagenetwork traffic. The connections to the network by the HMD 102,controller 104, and camera 108 may be wired or wireless. In someembodiments, content executed on the HMD 102 or displayable on a display107, can be obtained from any content source 120. Example contentsources can include, for instance, internet websites that providedownloadable content and/or streaming content. In some examples, thecontent can include any type of multimedia content, such as movies,games, static/dynamic content, pictures, social media content, socialmedia websites, etc.

As will be described below in more detail, a player 100 may be playing agame on the HMD 102, where such content is immersive 3D interactivecontent. The content on the HMD 102, while the player is playing, can beshared to a display 107. In one embodiment, the content shared to thedisplay 107 can allow other users proximate to the player 100 or remoteto watch along with the user's play. In still further embodiments,another player viewing the game play of player 100 on the display 107may participate interactively with player 100. For example, a userviewing the game play on the display 107 may control characters in thegame scene, provide feedback, provide social interaction, and/or providecomments (via text, via voice, via actions, via gestures, etc.,) whichenables users that are not wearing the HMD 102 to socially interact withplayer 100, the game play, or content being rendered in the HMD 102.

FIG. 1B illustrates a head-mounted display (HMD), in accordance with anembodiment of the invention. As shown, the HMD 102 includes a pluralityof lights 200A-H, J and K (e.g., where 200K and 200J are located towardthe rear or backside of the HMD headband). Each of these lights may beconfigured to have specific shapes and/or positions, and can beconfigured to have the same or different colors. The lights 200A, 200B,200C, and 200D are arranged on the front surface of the HMD 102. Thelights 200E and 200F are arranged on a side surface of the HMD 102. Andthe lights 200G and 200H are arranged at corners of the HMD 102, so asto span the front surface and a side surface of the HMD 102. It will beappreciated that the lights can be identified in captured images of aninteractive environment in which a user uses the HMD 102.

Based on identification and tracking of the lights, the location andorientation of the HMD 102 in the interactive environment can bedetermined. It will further be appreciated that some of the lights mayor may not be visible depending upon the particular orientation of theHMD 102 relative to an image capture device. Also, different portions oflights (e.g. lights 200G and 200H) may be exposed for image capturedepending upon the orientation of the HMD 102 relative to the imagecapture device. In some embodiments, inertial sensors are disposed inthe HMD 102, which provide feedback regarding positioning, without theneed for lights. In some embodiments, the lights and inertial sensorswork together, to enable mixing and selection of position/motion data.

In one embodiment, the lights can be configured to indicate a currentstatus of the HMD to others in the vicinity. For example, some or all ofthe lights may be configured to have a certain color arrangement,intensity arrangement, be configured to blink, have a certain on/offconfiguration, or other arrangement indicating a current status of theHMD 102. By way of example, the lights can be configured to displaydifferent configurations during active gameplay of a video game(generally gameplay occurring during an active timeline or within ascene of the game) versus other non-active gameplay aspects of a videogame, such as navigating menu interfaces or configuring game settings(during which the game timeline or scene may be inactive or paused). Thelights might also be configured to indicate relative intensity levels ofgameplay. For example, the intensity of lights, or a rate of blinking,may increase when the intensity of gameplay increases.

The HMD 102 may additionally include one or more microphones. In theillustrated embodiment, the HMD 102 includes microphones 204A and 204Bdefined on the front surface of the HMD 102, and microphone 204C definedon a side surface of the HMD 102. By utilizing an array of microphones,sound from each of the microphones can be processed to determine thelocation of the sound's source. This information can be utilized invarious ways, including exclusion of unwanted sound sources, associationof a sound source with a visual identification, etc.

The HMD 102 may also include one or more image capture devices. In theillustrated embodiment, the HMD 102 is shown to include image captureddevices 202A and 202B. By utilizing a stereoscopic pair of image capturedevices, three-dimensional (3D) images and video of the environment canbe captured from the perspective of the HMD 102. Such video can bepresented to the user to provide the user with a “video see-through”ability while wearing the HMD 102. That is, though the user cannot seethrough the HMD 102 in a strict sense, the video captured by the imagecapture devices 202A and 202B can nonetheless provide a functionalequivalent of being able to see the environment external to the HMD 102as if looking through the HMD 102.

Such video can be augmented with virtual elements to provide anaugmented reality experience, or may be combined or blended with virtualelements in other ways. Though in the illustrated embodiment, twocameras are shown on the front surface of the HMD 102, it will beappreciated that there may be any number of externally facing cameras ora single camera can be installed on the HMD 102, and oriented in anydirection. For example, in another embodiment, there may be camerasmounted on the sides of the HMD 102 to provide additional panoramicimage capture of the environment.

FIG. 2 illustrates one example of gameplay using the client system 106that is capable of rendering the video game content to the HMD 102 ofuser 100. In this illustration, the game content provided to the HMD isin a rich interactive 3-D space. As discussed above, the game contentcan be downloaded to the client system 106 or can be executed in oneembodiment by a cloud processing system. Cloud gaming service 112 caninclude a database of users 140, which are allowed to access particulargames, share experiences with other friends, post comments, and managetheir account information.

The cloud gaming service can also store game data 150 for specificusers, which may be usable during gameplay, future gameplay, sharing toa social media network, or for storing trophies, awards, status,ranking, etc. Social data 160 can also be managed by cloud gamingservice 112. The social data can be managed by a separate social medianetwork, which can be interfaced with cloud gaming service 112 over theInternet 110. Over the Internet 110, any number of client systems 106can be connected for access to the content and interaction with otherusers.

Continuing with the example of FIG. 2, the three-dimensional interactivescene viewed in the HMD can include gameplay, such as the charactersillustrated in the 3-D view. One character, e.g. P1 can be controlled bythe user 100 that is wearing the HMD 102. This example shows abasketball scene between two players, wherein the HMD user 100 isdunking a ball on another character in the 3-D view. The other charactercan be an AI (artificial intelligence) character of the game, or can becontrolled by another player or players (Pn). User 100, who is wearingthe HMD 102 is shown moving about in a space of use, wherein the HMD maymove around based on the user's head movements and body positions. Thecamera 108 is shown positioned over a display screen in the room,however, for HMD use, the camera 108 can be placed in any location thatcan capture images of the HMD 102. As such, the user 102 is shown turnedat about 90 degrees from the camera 108 and the display 107, as contentrendered in the HMD 102 can be dependent on the direction that the HMD102 is positioned, from the perspective of the camera 108. Of course,during HMD use, the user 100 will be moving about, turning his head,looking in various directions, as may be needed to take advantage of thedynamic virtual scenes rendered by the HMD.

FIG. 3A illustrates an example of a user wearing an HMD 102, where acamera 108, set at a fixed location, is directed toward the user. In oneembodiment, the camera 108 will have position that will enable viewingof the user, so that the HMD 102 and the controller 104 can be seen inthe video frames captured. In one example, the controller 104 mayinclude a marker, such as an LED light, which can be tracked at the sametime the HMD 102 is tracked. In the side view, the camera 108 iscapturing the side of the HMD 102. For purposes of illustration, the LEDmarkers are numbered as markers 1, 2, 3, and 4, for those on the frontunit 102 a. The LED markers on the band adjustment unit 102 b are notshown in the side view, but the top view shows that the camera 108 willbe able to capture at least markers 2 and 4 from the front unit 102 aand marker 6 from the band adjustment unit 102 b. The separation betweenthe front markers and the rear markers is, in one embodiment, estimated.

As described above, once the side view of FIG. 3A is detected in thecaptured video, based on analysis of the video frames, a calibration ofan estimated distance d1 is performed. If calibration has not occurredyet for a present session of use, a calibration will be performed,wherein several video frames are analyzed and compared to actual motiondata of the HMD, as obtained from HMD inertial sensors. The motion datain combination with the image data produces an estimation for thedistance d1. In one embodiment, as discussed above, the HMD isassociated with a geometric model, which is digital information as towhere each of the markers are located and shapes.

Using the calibration data, the geometric model is updated with thecalculation of what the estimated separation d1 is, for the currentsession of use. As the user continues to pay or interact with the HMD102, as shown in FIG. 3B, the distance data is utilized to track themovement and spatial positioning in space of the HMD and its tilts andangles relative to the camera 108. In FIG. 3B, for example, theillustration shows in the top view, that the user has turned his headback toward the camera 108. At such point in time, the camera may beable to see all of markers 1-6 on the front unit 102 a. The separationd1, in this example, assumes that the user may have adjusted theheadband 102 c at some point, such that the HMD would fit snug on theuser's head. A snug setting is preferred, as this avoids having the HMD102 move on the user's face. Further, movement of the HMD 102 also, inone embodiment, changes the scene view point into the scene as viewedthrough the HMD 102.

If adjustments are made to the headband 102 c, for example, the actualseparation between the front markers and the rear markers will change,which may (depending on the amount of adjustment) cause a disconnectbetween where the markers are anticipated by the geometric modelutilized by code of a game or program (e.g., library file(s)), andmovements of the HMD 102 may not translate well to changes in therendered scene in the HMD 102. For instance, movement of the HMD 102while only the front markers (i.e., LEDs) 1-4 are being tracked will notchange, as the geometric model of the HMD 102 stays static, as thosemarkers are at fixed positions. However, when the camera moves betweenthe front LEDs to the rear LEDs, if an adjustment in the headbandoccurred (to make it larger or smaller), the actual separation may notmatch to the geometric model or the calibrated geometric model. In suchcases, the system can detect the need for calibration or recalibrationif changes occurred during a session or some change or bend in theheadband moves the actual separation.

FIGS. 3C-1, 3C-2 and 3C-3 illustrate an example of a user that iswearing an HMD 102 that is strapped to the user's head via a headband102 c, the headband is connected to the front unit 102 a and the bandadjustment unit 102 b. In one embodiment, the band adjustment unit 102 bincludes an adjuster. The adjuster, in one example can be a rotary wheelthat allows tightening (making the size smaller) or lessening (makingthe size larger) of the headband. Other adjusters are also possible, asno limitation should be made to the use an example rotary wheel.

Once the headband is adjusted, the user may start to use the HMD in asession. In this example, the user is facing the camera 108, as shownfrom the top view. While the user is interacting in the session, thecamera is used to track the movement and positions of the HMD 102. Asnoted above, inertial sensors are also used during the tracking of theHMD 102, and the inertial data can be used to augment the opticaltracking by the camera 108. Still further, the inertial sensors are usedduring calibration, wherein image data is analyzed in conjunction withthe identified markers in the video images.

For example, the inertial sensors are generating position data, rotationdata, and general movement data of the HMD 102, which in someembodiments is generated multiple times during the generation of onevideo frame. Because the generated inertial data is so rich and has ahigh degree of fidelity, it is possible to estimate a distance betweenidentified image markers or objects over a period of several frames. Assuch, the calibration will produce an estimation of the actual distancebetween markers, e.g., between front and rear LEDs of the HMD 102. Inthe example captured frames 302, it is shown that the front of the frontunit 102 a of the HMD 102 is visible.

FIGS. 3D-1, 3D-2 and 3D-3 illustrate an example of where the user hasrotated his head about 45 degrees away from the start position shown inFIG. 3C. It should be understood that rotation need not be perfectrotation, as a normal human movement of rotation can include movement ofthe face up and down (e.g., tilt), in combination with rotation. Forpurposes or example, simple rotation is shown, and it is assumed thatother motion also occurred (e.g., tilts up/down at different points intime during the rotation). In action, the movement occurs over someperiod of time, which may happen (at any point in time) in response tothe user wishing to explore a different scene in a game or viewdifferent content in a virtual space or view different scene regionswithin a movie or video. At one point in time, as shown in FIG. 3D-3,the captured frames 303 will include image data that includes markers 2,4 and 6. For example purposes, suppose that this is the first time therear marker 6 is captured in the frames, as the image data is analyzed.If calibration has not occurred for the current session, or it isdetermined that time has passed since the last calibration, the systemwill perform a calibration operation, wherein the distance between thefront LEDs and the rear LED is calculated.

As discussed above, this is done by analyzing rotation data andpositional data from the inertial sensors (e.g., gyroscope andaccelerometer) of the HMD 102. This analysis will render an estimatedseparation distance, which is used for further tracking. In oneembodiment, this initial estimated distance is saved to the geometricmodel of the HMD 102, which contains information regarding spatialseparations and geometric shapes of the LEDs and other characteristicsof the modeled HMD 102. In other embodiments, the calibration thatincludes the estimated distance is used until further examinationindicates that a change has occurred (i.e., a new examination yields adifferent separation distance than the one used in the currentcalibrated geometric model).

This change may have occurred when the user adjusted the headband, ortook off the HMD 102 and another user adjusted and tried on the HMD 102,or the headband 102 got bent or moved during the session. If the newestimated separation distance is beyond a predefined margin, the newlyestimated separation distance is added to the calibrated geometricmodel. As mentioned above, the calibrated geometric model can be used byan application and/or the system to provide accurate tracking of the HMD102 movements and positions and to provide content to the HMD 102consistent with the direction, movement and motions made by the userwearing the HMD 102.

By maintaining the calibration of the geometric model updated to anyidentified change in distance or actual adjustment in the headband size,the content provided to the HMD 102 is prevented from seeing jumps, popsor inaccurate jitter when displayed in the HMD 102, especially when theuser moves the HMD 102 to the side (i.e., away from substantially normalto the camera 108).

FIGS. 3E-1, 3E-2 and 3E-3 illustrate an example where the user has movedto about 90 degrees away from the position shown in FIGS. 3C-1 and 3C-2.At this point, if no adjustment has been made, and the calibrationappears to be within margin (as compared to further backgroundcalibrations, but unapplied to the model), then the system will continueto use the estimated distance between the front and rear LEDs forrendering content to the HMD 102.

FIGS. 3F-1, 3F-2 and 3F-3 illustrate a time in the session where theuser has rotated about 180 degrees from the position of FIG. 3E-1. Inthis example, the camera 108 is now producing captured frames 302wherein markers 1, 3 and 5 are visible. As noted above, the originalcalibration of the separation distance may be used continually, until anext session, until the user stops moving and then resumes or when theuser adjusts the headband. In still another embodiment, the system canagain notice that the front and back LEDs are viewable, but also noticethat it is the opposite side that is now viewable. In other words, theviewable LEDs of FIG. 3E-3 are the opposite side LEDs from thoseviewable in FIG. 3F-3.

Thus, the system can require that calibration be made for each sideindependently. Thus, although the side in FIG. 3E-3 was alreadycalibrated and an estimated distance separation is being used in thegeometric model, the system can independently run a calibration of theother side and estimate another separation distance that is added to thegeometric model. In this manner, separate calibrations for both sidescan cure situations where an adjustment is made to one side of theheadband and not the other side of the headband, or if a kink in theheadband produces different actual separations between the rear LED andthe front visible LEDs.

FIGS. 3G-1, 3G-2 and 3G-3 illustrate an example where the back of theHMD 102 is visible in the captured frames 302. In this example, thetracking of the LEDs 5 and 6 can occur using the original geometricmodel of the HMD 102, as the LEDs 5 and 6 are on a fixed structure thatdoes not change. In other embodiments, if the back LEDs were not fixedrelative to each other, it is also possible to calibrate the one of therear LEDs to the front LEDs and then the calibrated back LED to theother LED, as the user's head rotates around at different times.

FIG. 4A illustrates a top view of the HMD 102, and a camera 108 which ispositioned to be able to identify markers (LEDs) associated with the HMD102. The HMD 102 top view shows the front unit 102 a and the bandadjustment unit 102 b connected by way of the headband 102 c. The frontunit 102 a includes the optics block section of the HMD 102. The opticsblock is configured to be adjustable by an adjustment gap 402. As such,the optics block can move toward and away the face of the user whenattached to a user's head. In one embodiment, the adjustment gap 402 ismade adjustable for comfort of the HMD 102 user and also to provideadditional space between the user's eyes and structure in the opticsblock. The additional space can be utilized to accommodate eyeglasses ofthe user. As shown, the band adjustment unit 102 b includes a rotarywheel 404, which is usable to adjust the headband in a manner thateither contracts into the band adjustment unit 102 b or expand out ofthe band adjustment unit 102 b.

As noted above, other adjusters are also possible, as no limitationshould be made to the use an example rotary wheel. For instance, otheradjusters can include belt adjusters, notched adjusters, pin and holesadjusters on a band, snap-in or snap together adjusters, Velcroadjusters, tape adjusters, belt and buckle adjusters, rope adjusters,link connector adjusters, elastic adjusters, or combinations of varioustypes of adjusters and clips and locks.

Adjustments are made to provide for a snug or comfortable fit by a userwho has placed the HMD on his or her head. Because of thisadjustability, the actual separation distance D of the HMD 102 fromfront to back will vary, based on the adjustments made. For illustrationpurposes, if the optics block were pushed toward the users facecompletely and the rotary wheel 404 was adjusted to bring the size ofthe headband to its smallest possible size, the separation distancewould be dC. However, given that customize adjustments by the users, avariation of separation dA will be present for adjustments in the opticsblock, and also for adjustments in the band adjustment unit 102 b byseparation dB. At any one time, the actual physical separation D betweenthe LEDs in the front unit of the optics block and the rear LEDs in theband adjustment unit 102 b will vary, and for this reason calibration isperformed as described above to identify an estimated separationdistance between the front LED and the rear LED, based on the currentadjustments.

As illustrated, the separation distance is equal to: D=dC+/−dA+/−dB. Thecalibration during use of the HMD will therefore identify a closeapproximation estimate of the actual separation between the visible rearLED and the visible front LEDs, therefore defining the separationdistance. The separation distance is then added to the geometric modelof the HMD 102, which is utilized to render content to the HMD based onthe position, rotation, movement and orientation of the HMD during use.

FIG. 4B illustrates an example of a top view of an HMD 102 where aseparation d1 is physically set for the physical HMD. The physicalsetting can occur due to adjustments made to the headband 102 c of theHMD. The video frame 410, when analyzed will show that a separationdistance d1 is estimated, based on analysis of multiple frames andsimultaneous analysis of the inertial data produced by sensors in theHMD 102.

In comparison, FIG. 4C illustrates a top view of the HMD 102 after anadjustment is made to make a larger headband 102 c. This process willalso include analysis of the captured video frames to identify thecurrent positions of the rear and front LEDs, and then run through acalibration operation to determine if the resulting estimated separationis the same as previously calibrated for the size of FIG. 4B. In oneembodiment, the system will determine that the size is now larger by aΔd, in other words d1+Δd, which is now d2. If the difference Δd isgreater than a margin of air acceptable to the system, the originalcalibration will be maintained.

If the new estimated distance d2 is different than estimated distanced1, then the calibration data in geometric model is updated to includethe distance d2. In another embodiment, instead of relying on anadditional calibration step to identify whether a new calibration isproducing an estimated distance larger than the original estimateddistance, an adjustment of the rotary wheel 404 or any other adjustmentmechanism can trigger a signal, such as a flag or data indicative of achange to the size of the headband 102 c.

As a result, this signal can be communicated back to the system whichwould then enforce that a new calibration be performed to identify a newestimated separation between a front LED and a rear LED. In still otherembodiments, recalibration can be performed when the system determinesthat inactivity or non-movement of the HMD has occurred for some time.This would be an indicative signal that new calibration data for thedistance between the front LED and the rear LED is necessary, as it ishighly likely that a new user has put the HMD 102 on to begin a newsession or to participate in an existing session.

For instance, during the session enjoyed by a first player, a nearbysecond player may wish to view the content being rendered in the HMD andthe first player can or may remove the HMD so the second player can viewthe content for a small period of time. During this sharing, it ispossible that the HMD headband may have been adjusted. In fact, it ispossible that to adjustments occurred: a first adjustment for the secondplayer to temporarily wear the HMD, and a second adjustment so that thefirst player can resume the session after obtaining the HMD back fromthe second player.

FIG. 4D provides an example flow diagram of operations that can beperformed to identify the separation distance between the front markerson the HMD and the visible rear marker associated with the headband ofthe HMD. In operation 420 includes capturing video frames of a headmounted display when worn by a user who is interacting with multimediacontent. The multimedia content, as described above can include varioustypes of media. The types of media can include video games, movies, webcontent, video clips, audio content, combinations of video and audio,video conferencing, social media interactive communications, and/orcombinations thereof.

During the time when the user is interacting with the multimedia whilewearing the HMD, content is being provided to the HMD for rendering andviewing by the user. In one embodiment, a camera is also provided toview the position and orientations and movements of the HMD. The camera,in one embodiment produces multiple images in the form of video frames.The video frames can then be examined to identify markers disposed onthe HMD, in operation 422. The markers, as mentioned above, can includeLEDs.

In other embodiments, the markers can include markings that do notilluminate, shapes, retro reflective surfaces, different color surfaces,different color regions or surfaces, combinations of LEDs and surfaces,combinations of LEDs, retro reflective tapes, reflective paints,reflective surfaces, numerical data, QR code data, patterns, images, andcombinations of two or more thereof. As such, although one embodimentutilizes LEDs that light up a region of the HMD with a color, that colorcan vary and the shape and orientation of the LED surface or region thatis illuminated can also vary. In some embodiments, multiple differentcolors can be used by the HMD, such that certain colors identifypositions. In yet other implementations, different colors can beutilized on different parts of the HMD or on different HMD's whenmultiple HMD's are being tracked. If multiple HMD users are beingtracked, different colors can be assigned to different players. The LEDscan also be configured to illuminate in patterns, intermittently, orpulsate to communicate codes or data by way of the image data capturedand analyzed by the system.

In some embodiments, the LEDs can be configured to change inillumination intensity. The change in illumination can be dynamicallycontrolled, such as in response to detecting environmental lighting. Ifthe lighting in a room is bright, the illumination can be intensified ordifferent colors can be selected to improve or enhance tracking. In someother embodiments, the colors selected can be ones that are different orprovide more contrast to surrounding colors of a room, environment, theperson's clothes, or combinations thereof.

In operation 424, the identified markers can be tracked by analyzing theimage data captured by the camera. Tracking of the HMD is facilitated bythe markers identified on the front optics block, which are oriented inan arrangement that enables identification of position and orientationof the front optics block of the HMD. In the illustrated example, themarkers are positioned at the front corners of a substantiallyrectangular optics block, when viewed directly on from the camera 108.Thus, when the user moves his head down, the two upper LEDs and the twobottom LEDs appear to become closer in the images captured by thesystem, upon analyzing the image data in the video frames. Similarprocessing can be made to determine when the user is turning to theleft, turning to the right, tilting to the left, tilting to the right,or combinations of movements such as rotations and tilting. In oneembodiment, operation 426 determines if a marker on the band adjustmentunit is visible in the captured video frames.

As noted above, the band adjustment unit is disposed substantiallytoward the back of the users head when the HMD is used and worn. Assuch, the marker on the band adjustment unit will be visible at a pointin time when at least two of the front markers are also visible. At thispoint, a calibration operation can occur to identify the distancebetween the front LEDs and the rear LEDs in operation 428. In thisoperation, at least one of the frames are examined and inertial data isexamined to estimate a distance separating the visible markers on thefront optics block and a visible marker on the band adjustment unit. Asnoted above, multiple frames can be analyzed to identify movement of theHMD and changes in the movement in the image data, which are mappedtogether to identify an estimated separation distance.

In one embodiment, inertial data, which can include gyroscope data andaccelerometer data, is associated to different points in time at whichthe various frames show movement of the LEDs, relative to other frames.In some implementations, utilized gyroscopes can provide samples at arate that is many times per frame, which is a rich set of data thatallows for very accurate estimation of the amount of movement occurringbetween the front and rear LEDs, which enable an accurate estimation ofthe separation distance. In some embodiments, the accelerometer canprovide samples at a rate of up to about 2000 frames per second, andthis information is also usable to identify the position and rotation ofthe HMD.

In one embodiment, operation 430 is processed to determine if a flag hasbeen detected, which indicates a change in size to the band adjustmentunit. For example, a circuit in the band adjustment unit can trigger asignal, a flag, or communicate data indicative of a change and/or anamount of change made to the headband. If the flag has not beendetected, the operation moves to 432 where the HMD use can continue toexamine additional frames and inertial data to determine if an update isnecessary to the estimated distance, while keeping the estimatedseparation distance in place and associated with the geometric model ofthe HMD. If it is detected that a flag has been set in operation 430,the method moves to operation 434 were it is assumed that a differentdistance in the headband has been set, which will require a new estimateof the separation distance. Once the new separation distance isestimated, the new separation distance is added to the three-dimensionalmodel of the HMD 102, which is then used during the session, nowsession, or for further use of the HMD.

FIG. 4E illustrates another flow diagram example, utilized to makecalibrations to the separation distance between LEDs of the front unitof the HMD 102 and an LED associated with a rear portion of the HMD. Inthis example, video frames of the HMD are examined to identify markersused for optical position and orientation tracking in operation 440. Inoperation 442 it is determined if the front and back markers are visiblein the video frames. If they are not visible, the method continues toidentify and use the markers on the front unit of the HMD for tracking,as the markers on the front unit do not change in separation due totheir fixed orientation on the front unit. Because they are fixed on thefront unit, the geometric model of the HMD is consistent with the knownseparation on the front unit of the HMD.

In operation 444, once the front and back markers are visible in atleast one frame, the operation performs an estimate of the distancebetween the front marker and the rear marker of the HMD from the videoframe data and HMD inertial data. The estimate of the separation is madeby examining video frame data and correlating it to inertial data forwhen the user moves his or her head, thus moving the location of thevisible LEDs at a particular rate, orientation, and distance. Once theestimated distances are determined, that estimated distance separationis added to the original geometric model of the HMD.

By adding the estimated distance to the model, the model has now beencalibrated to the estimated separation distance between the frontvisible and rear visible markers. In operation 446, it is determined ifthe session has ended. It can be determined that the session has endedwhen the HMD does not move for a period of time, the HMD is placed onthe surface and has not moved for some time, the HMD is no longer shownmoving with the user, who may be detected in the image data, a sensorindicator generates a flag indicating the end of the session, the gamecode has ended interactivity with the game, the user has turned off theunit, the user has placed the unit in sleep mode, or the user isinteracting with display data or navigating screens or settings that donot require motion tracking of the HMD, etc. Accordingly, when thesession has ended, it is determined that the distance between the frontand rear markers must be updated for the next user or next session.

If the session has not ended as determined by operation 446, adetermination is made in operation 448 to determine if a flag has beenset indicative of the headband being adjusted. If the headband has beenadjusted and a flag is set, the system will request that the distancebetween the front and rear markers should be updated for a next sessionor continued play or use. If the flag is not set in operation 448, thesystem can proceed to refine the estimated distance during use. As notedabove, this can occur periodically or every time the user exposes theside of the HMD to the camera after only exposing the front LEDs to thecamera.

In some embodiments, the estimated distance can be calculated, but thecalibration is not automatically updated to the geometric model. In thisexample, calibration can continue to occur continuously until one of thecalibrations identifies an estimated distance that is substantiallydifferent or beyond the margin allowable, so that the new calibratedestimated distance should be added to the geometric model for continueduse.

In one embodiment, it has been determined that a change in the estimatedseparation distance up to about 50 mm is within the margin (or up toabout 30 mm is within the margin, or up to about 20 mm is within themargin, or up to about 15 mm is within the margin, or up to about 10 mmis within the margin, or is up to about 5 mm is within margin), and willnot adversely affect the tracking and the delivery of multimedia contentto the HMD. In some embodiments, if the estimated separation distancehas changed by more than about some predefined mm value (e.g., 1 mm, 2mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, 10 mm, 15 mm, 20 mm,etc.), then a new estimated separation distance should be added to thegeometric model, to avoid inconsistencies in the track positions versusthe position that the software and system expects the HMD to be in. Whensuch inconsistencies occur, it is possible that the images or contentrendered on the HMD can show a glitch, a pop, or some sort of jump thatis not normal. If the difference in the estimated separation distance isminimal, less than a predefined mm value (e.g., 1 mm, 2 mm, 3 mm, 4 mm,5 mm, 6 mm, 7 mm, 8 mm, 9 mm, 10 mm, 15 mm, 20 mm, etc.), then it isdetermined that no update to the geometric model is necessary as theimages will render without noticeable loss of quality.

In some embodiments, the margin or amount by which the separationdistance changes can vary depending on the content being displayed inthe HMD 102. If the content requires more precision, such as in complexor very rich graphically intensive games, the margin may be lower. Ifthe content is Web data, webpages, or other content that does not changeoften or is substantially fixed or less interactive, the margin can belarger.

FIG. 4F illustrates yet another embodiment, wherein examination of videoframes and inertial data is performed to identify separation distancesbetween the front and rear markers of the HMD. In this example,operation 460 identifies a geometric model of the HMD and markers on theHMD. The markers on the HMD include front markers and rear markers. Asmentioned above, the front markers are fixed at a set orientationrelative to the front unit of the HMD. The rear markers are coupled to aheadband, which is adjustable. Therefore, by adjusting the headband, therear markers will be adjusted further or closer to the markers of thefront unit of the HMD.

The separation distance between the two must be estimated, so that thedistance can be calibrated to the geometric model of the HMD. Inoperation 462, video frames are captured of the HMD when worn by theuser. In operation 464, it is determined if the front and rear markersare visible in video frames captured of the HMD during a session orduring use. If they are not visible, the method continues to capturevideo frames of the HMD in operation 462, utilizing position andorientation information from the markers disposed on the front unit ofthe HMD, for which the separation distances and relative orientationsand shapes are geometrically known.

In operation 466, it is determined that one or more frames of the HMDmust be examined while examining accelerometer data and gyroscope datafrom the HMD. The one or more frames of the HMD captured by the camerawill then be associated to the accelerometer data and gyroscope data. Byanalyzing successive frames and associated image frames in conjunctionwith the inertial data, a separation distance is estimated between thefront markers and the rear marker that is visible in the image frames.In operation 468, it is determined if calibration has been performed. Ifcalibration has not yet been performed the method moves to operation472. In operation 472, the geometric model of the HMD is calibrated.

As such, the geometric model will include the estimated distance betweenthe front and rear markers, as determined in operation 466. If it isdetermined that calibration has occurred in the past, the method movesfrom operation 468 to operation 470. In operation 470 it is determinedif the calibration is accurate. As noted above, the calibration maybecome in accurate if an adjustment to the headband has been made, akink in the headband has been introduced, or the HMD has been sharedwith another user who has adjusted the headband or manipulated that HMDto produce a different estimated separation.

If the different estimated separation has been determined by examiningagain the video frames when the front and rear markers are visible, andcomparing them to the inertial data, and the new estimated separationdistance is greater than an allowed margin, the calibration is updatedin operation 472. If the calibration remains accurate, or is within themargin, then the calibrated geometric model of the HMD will continue tobe used with the originally determined separation distance between thefront and rear markers.

Therefore, the calibrated geometric model will continue to be used todetermine the position and orientation of the HMD when tracked and whenpresenting multimedia content which changes at least by movement and/orposition of the HMD by the user. As noted above, the method can continueto perform calibrations that are not added to the geometric model, butare used to determine when the separation distance needs to be updatedin response to some change or adjustment to the headband of the HMD.

FIG. 5A provides an example of a top view of the HMD 102, in accordancewith one embodiment of the present invention. In this example, it isshown that the optics block associated with the front unit 102 a of theHMD includes various components. Without reciting all of the componentscontained within the optics block, some components that are useful fortracking and determining the estimated separation distance between thefront and rear LEDs are shown, by way of example.

In block 502, example components can include an accelerometer 510, agyroscope 512, and LED control circuit 514, an image processing circuit516, optics 518, display screens 520, and other components and circuitryused for rendering content to the HMD, when communication is made with aclient system 106. As mentioned above, the HMD can be communicatingdirectly with a client system 106 locally, and the connection can bemade by way of a wire or wireless connection. In one embodiment, the HMDincludes a circuit for communicating with the HMD. The circuit can befor transmitting and receiving signals and/or data to and from thecomputing system. The circuit, in a wireless configuration, can includea wireless communication link. The circuit, in a wired configuration,can include cable connectors and interfaces for enabling plugging,connecting, extending and/or interfacing between the HMD and thecomputing system.

In some implementations, the user of the HMD 102 can also be interactingwith the content rendered in the HMD using a controller 104 or anotherinput technology. In another embodiment, the HMD 102 can includecircuitry for communicating directly with a router or networkingcomponents that allow communication with a cloud infrastructure.

As is further shown, the HMD 102 can include an additional headband,which is defined by a headband portion 102 d. The additional headbandcan include padding, and can be adjusted separately from the headband102 c. The top view of the HMD 102 is provided with the illustration ofa user's head which has the HMD positioned thereon. Further illustratedis optional components 504, which may be integrated or contained withinthe band adjustment unit 102 b. The components can include, in someimplementations, components shown in block 504 a.

Block 504 80 includes an accelerometer 522, a gyroscope 524, a headbandadjustment detector 526 that is configured to generate a flag 530. Theflag 530, in one embodiment, is set when a detection is made that anadjustment was made to the headband, such as by adjusting the rotarywheel 404. In another embodiment, block 504 b can be disposed within theband adjustment unit 102 b. In this example the band adjustment unit 102b includes no circuitry, but simple wiring that connects the circuitrydisposed in the optics block (502) to the LEDs 5 and 6.

In still another embodiment, block 504 c can include an accelerometer522 and optionally a headband adjustment detector 526, which is utilizedto generate the flag 530. Accordingly, it should be understood that therear section of the headband, which is referred to herein as the bandadjustment unit 102 b can include no circuitry, or can includeadditional circuitry to allow further determinations of position byvirtue of output from the inertial sensors.

FIG. 5C illustrates an example of a side view of the user wearing HMD102, wherein the front unit 102 a is disposed over the face and eyes ofthe user while being connected to the headband 102 c and the bandadjustment unit 102 b. FIG. 5C illustrates a side view of the HMD 102 toshow the relative positioning of the optics block that includes thefront unit 102 a and the positioning of the band adjustment unit 102 b,which is connected to the headband 102 c.

As shown, when the HMD 102 is positioned over a users head, thepositioning of the optics block and the band adjustment unit 102 b willhave a substantially known relative orientation. If the band adjustmentunit 102 b is configured to include a rear accelerometer and agyroscope, data can be collected from both the rear of the HMD and thefront of the HMD, which already includes a front accelerometer and agyroscope. In one embodiment, data obtained from the inertial sensorsfrom the front of the HMD and from the rear HMD can be plotted todetermine a relative positioning of the components, such as relative toa center point of the HMD (e.g. an approximate center of where the usershead would reside).

For example, the front accelerometer and the rear accelerometer cangenerate data regarding the rotational forces and angles of rotationrelative to a gravitational force. In a substantially normal or restingposition, the optics block will be substantially parallel to thegravitational forces determined by the front accelerometer. In the sameposition, the rear accelerometer may experience an angle of rotationrelative to the gravitational forces. The position and vocational datareceived from the front and rear accelerometers can therefore be mappedor monitored to determine when an adjustment in the headband has beenmade, which would require recalibration of a separation distance betweenthe front LEDs and the rear LED. In another embodiment, by examining therelative inertial data between the front and the rear LEDs, theseparation distance is estimated by simply using the inertial data ofthe front and rear sections of the HMD, without having to rely onexamination of the video frames captured by the camera.

Thus, it is possible to examine the pitch and the tilt between the frontand rear accelerometer data to identify an approximate center pointseparation between the front and rear sections of the HMD. Adjustmentsof the headband would therefore change the distances to the center,which would then signal a need to adjust the estimated separationdistance in the geometric model of the HMD, to allow accurate trackingof the HMD during use. Furthermore, by including a gyroscope in thefront and in the back, errors in gyroscope data can be reduced, wherebydifferences can be canceled out or refined over time.

Therefore, in some implementations, if there is an accelerometer insidethe rear headband, it is possible to determine the tilt of the rear vs.the front. This could signal, for example, that there may be a flex inthe band. This information can then be communicated to the user by wayof a recommendation, e.g., such as by a message in the screen of theHMD.

Still further, with the front and rear accelerometers, plus a gyro, itis possible to estimate the distance the headband is away from the frontaccelerometer. This is possible because the back accelerometer willexperience a larger centripetal acceleration (e.g., since it istraveling on a longer “arm” than the front one). And, because you knowhow fast the HMD is rotating (e.g., with a gyro) it is possible todetermine or estimate how far away the accelerometers are away from eachother. This information can be used to determine an estimated separationdistance, which can then be added as calibration to the geometric modelof the HMD 102, and which is used for tracking during rendering ofcontent.

Furthermore, since both an accelerometer and gyro are disposed on therear headband, it is possible to run two independent tracking “devices”,and average out the results, or perform additional estimation orblending. In one embodiment, this estimated separation between the frontand rear LEDs would be the difference between the LEDs. In furtherembodiments, if both sides of the HMD are estimated for separationbetween the front and rear LEDs (e.g., independently), it is possible toblend or fuse together the results to arrive at a more optimal estimatedseparation distance for use on both sides of the geometric model.

FIGS. 6A-6C illustrate various views of the HMD 102, when worn on a headof a human user, in accordance with one embodiment. In FIG. 6a , it isshown that LED 2 can wrap around the front face and side face of the HMD102. The same can occur for LED 4. In this way, the LEDs 2 and 4 arevisible both from the front and from the side. When the user turns tothe side, LED 6 would also be visible, and so would at least one of LEDs2 and/or 4 (e.g., depending on the angle, tilt, orientation, and/orposition). The rear section 102 b is also shown connected to theheadband 102 c.

FIG. 6B shows the HMD 102 when the front unit is facing the camera. Inthis example, if the user if facing down, it is possible that at timesonly LEDs 1 and 2 would be visible. However, if the user locks up, LEDs3 and 4 would be visible with LEDs 1 and 2, depending on the orientationof the user's head with respect to the camera 108. FIG. 6C shows a topview, wherein the LEDs 1 and 2 and 6 and 5 may be viewable. Also shownis the adjuster on the rear section 102 b. Further shown is the headbandportion 102 d, which may be a second headband used to provide additionalsupport and comfort. Headband portion 102 d has its own adjuster, yetthat adjuster may not usually affect the spatial physical position ofthe rear LEDs relative to the front LEDs.

FIG. 7 shows a side view of the HMD. In this example, it is shown thatthe rear section 102 b, which has the adjuster 404, can sometimes bepulled on or can twist. Such changes can cause adjustments in thecalibrated positioning of the rear LEDs relative to the front LEDs. Insuch a case, adjustment or updates to the calibrated separation distancecan be made, and the geometric model of the HMD updated as well.

FIG. 8 illustrates a user wearing the HMD 102, during use, in accordancewith one embodiment. In this example, it is shown that the HMD istracked 802 using image data obtained from captured video frames by thecamera 108. Additionally, it is shown that the controller can also betracked 804 using image data obtained from captured video frames by thecamera 108. Also shown is the configuration where the HMD is connectedto the computing system 106 via a cable 806. In one embodiment, the HMDobtains power from the same cable or can connect to another cable. Instill another embodiment, the HMD can have a battery that isrechargeable, so as to avoid extra power cords.

With reference to FIG. 9, a diagram is shown illustrating examplecomponents of a head-mounted display 102, in accordance with anembodiment of the invention. It should be understood that more or lesscomponents can be included or excluded from the HMD 102, depending onthe configuration and functions enabled. The head-mounted display 102may include a processor 900 for executing program instructions. A memory902 is provided for storage purposes, and may include both volatile andnon-volatile memory. A display 904 is included which provides a visualinterface that a user may view.

The display 904 can be defined by one single display, or in the form ofa separate display screen for each eye. When two display screens areprovided, it is possible to provide left-eye and right-eye video contentseparately. Separate presentation of video content to each eye, forexample, can provide for better immersive control of three-dimensional(3D) content. As described above, in one embodiment, the second screen107 is provided with second screen content of the HMD 102 by using theoutput for one eye, and then formatting the content for display in a 2Dformat. The one eye, in one embodiment, can be the left-eye video feed,but in other embodiments it can be the right-eye video feed.

A battery 906 may be provided as a power source for the head-mounteddisplay 102. In other embodiments, the power source can include anoutlet connection to power. In other embodiments, an outlet connectionto power and a battery 906 may be provided. A motion detection module908 may include any of various kinds of motion sensitive hardware, suchas a magnetometer 910, an accelerometer 912, and a gyroscope 914.

An accelerometer is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis (e.g., six-axis)models are able to detect magnitude and direction of the acceleration indifferent directions. The accelerometer is used to sense inclination,vibration, and shock. In one embodiment, three accelerometers 912 areused to provide the direction of gravity, which gives an absolutereference for two angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic fieldin the vicinity of the head-mounted display. In one embodiment, threemagnetometers 910 are used within the head-mounted display, ensuring anabsolute reference for the world-space yaw angle. In one embodiment, themagnetometer is designed to span the earth magnetic field, which is ±80microtesla. Magnetometers are affected by metal, and provide a yawmeasurement that is monotonic with actual yaw. The magnetic field may bewarped due to metal in the environment, which causes a warp in the yawmeasurement. If necessary, this warp can be calibrated using informationfrom other sensors such as the gyroscope or the camera. In oneembodiment, accelerometer 912 is used together with magnetometer 910 toobtain the inclination and azimuth of the head-mounted display 102.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, threegyroscopes 914 provide information about movement across the respectiveaxis (x, y and z) based on inertial sensing. The gyroscopes help indetecting fast rotations. However, the gyroscopes can drift overtimewithout the existence of an absolute reference. This requires resettingthe gyroscopes periodically, which can be done using other availableinformation, such as positional/orientation determination based onvisual tracking of an object, accelerometer, magnetometer, etc.

A camera 916 is provided for capturing images and image streams of areal environment. More than one camera (optionally) may be included inthe head-mounted display 102, including a camera that is rear-facing(directed away from a user when the user is viewing the display of thehead-mounted display 102), and a camera that is front-facing (directedtowards the user when the user is viewing the display of thehead-mounted display 102). Additionally, a depth camera 918 may beincluded in the head-mounted display 102 for sensing depth informationof objects in a real environment.

The head-mounted display 102 includes speakers 920 for providing audiooutput. Also, a microphone 922 may be included for capturing audio fromthe real environment, including sounds from the ambient environment,speech made by the user, etc. The head-mounted display 102 includestactile feedback module 924 for providing tactile feedback to the user.In one embodiment, the tactile feedback module 924 is capable of causingmovement and/or vibration of the head-mounted display 102 so as toprovide tactile feedback to the user.

LEDs 926 are provided as visual indicators of statuses of thehead-mounted display 102. For example, an LED may indicate batterylevel, power on, etc. A card reader 928 is provided to enable thehead-mounted display 102 to read and write information to and from amemory card. A USB interface 930 is included as one example of aninterface for enabling connection of peripheral devices, or connectionto other devices, such as other portable devices, computers, etc. Invarious embodiments of the head-mounted display 102, any of variouskinds of interfaces may be included to enable greater connectivity ofthe head-mounted display 102.

A WiFi module 932 may be included for enabling connection to theInternet via wireless networking technologies. Also, the head-mounteddisplay 102 may include a Bluetooth module 934 for enabling wirelessconnection to other devices. A communications link 936 may also beincluded for connection to other devices. In one embodiment, thecommunications link 936 utilizes infrared transmission for wirelesscommunication. In other embodiments, the communications link 936 mayutilize any of various wireless or wired transmission protocols forcommunication with other devices.

Input buttons/sensors 938 are included to provide an input interface forthe user. Any of various kinds of input interfaces may be included, suchas buttons, gestures, touchpad, joystick, trackball, etc. An ultra-soniccommunication module 940 may be included in head-mounted display 102 forfacilitating communication with other devices via ultra-sonictechnologies.

Bio-sensors 942 are included to enable detection of physiological datafrom a user. In one embodiment, the bio-sensors 942 include one or moredry electrodes for detecting bio-electric signals of the user throughthe user's skin, voice detection, eye retina detection to identifyusers/profiles, etc.

The foregoing components of head-mounted display 102 have been describedas merely exemplary components that may be included in head-mounteddisplay 102. In various embodiments of the invention, the head-mounteddisplay 102 may or may not include some of the various aforementionedcomponents. Embodiments of the head-mounted display 102 may additionallyinclude other components not presently described, but known in the art,for purposes of facilitating aspects of the present invention as hereindescribed.

It will be appreciated by those skilled in the art that in variousembodiments of the invention, the aforementioned handheld device may beutilized in conjunction with an interactive application displayed on adisplay to provide various interactive functions. The exemplaryembodiments described herein are provided by way of example only, andnot by way of limitation.

In one embodiment, clients and/or client devices, as referred to herein,may include head mounted displays (HMDs), terminals, personal computers,game consoles, tablet computers, telephones, set-top boxes, kiosks,wireless devices, digital pads, stand-alone devices, handheld gameplaying devices, and/or the like. Typically, clients are configured toreceive encoded video streams, decode the video streams, and present theresulting video to a user, e.g., a player of a game. The processes ofreceiving encoded video streams and/or decoding the video streamstypically includes storing individual video frames in a receive bufferof the client. The video streams may be presented to the user on adisplay integral to client or on a separate device such as a monitor ortelevision.

Clients are optionally configured to support more than one game player.For example, a game console may be configured to support two, three,four or more simultaneous players (e.g., P1, P2, . . . Pn). Each ofthese players may receive or share a video stream, or a single videostream may include regions of a frame generated specifically for eachplayer, e.g., generated based on each player's point of view. Any numberof clients can be local (e.g., co-located) or are geographicallydispersed. The number of clients included in a game system may varywidely from one or two to thousands, tens of thousands, or more. As usedherein, the term “game player” is used to refer to a person that plays agame and the term “game playing device” is used to refer to a deviceused to play a game. In some embodiments, the game playing device mayrefer to a plurality of computing devices that cooperate to deliver agame experience to the user.

For example, a game console and an HMD may cooperate with the videoserver system to deliver a game viewed through the HMD. In oneembodiment, the game console receives the video stream from the videoserver system and the game console forwards the video stream, or updatesto the video stream, to the HMD and/or television for rendering.

Still further, the HMD can be used for viewing and/or interacting withany type of content produced or used, such video game content, moviecontent, video clip content, web content, advertisement content, contestcontent, gamboling game content, conference call/meeting content, socialmedia content (e.g., posting, messages, media streams, friend eventsand/or game play), video portions and/or audio content, and content madefor consumption from sources over the internet via browsers andapplications and any type of streaming content. Of course, the foregoinglisting of content is not limiting, as any type of content can berendered so long as it can be viewed in the HMD or rendered to a screenor screen of the HMD.

Clients may, but are not required to, further include systems configuredfor modifying received video. For example, a client may be configured toperform further rendering, to overlay one video image on another videoimage, to crop a video image, and/or the like. For example, clients maybe configured to receive various types of video frames, such asI-frames, P-frames and B-frames, and to process these frames into imagesfor display to a user. In some embodiments, a member of clients isconfigured to perform further rendering, shading, conversion to 3-D,conversion to 2D, distortion removal, sizing, or like operations on thevideo stream. A member of clients is optionally configured to receivemore than one audio or video stream.

Input devices of clients may include, for example, a one-hand gamecontroller, a two-hand game controller, a gesture recognition system, agaze recognition system, a voice recognition system, a keyboard, ajoystick, a pointing device, a force feedback device, a motion and/orlocation sensing device, a mouse, a touch screen, a neural interface, acamera, input devices yet to be developed, and/or the like.

A video source may include rendering logic, e.g., hardware, firmware,and/or software stored on a computer readable medium such as storage.This rendering logic is configured to create video frames of the videostream based on the game state. All or part of the rendering logic isoptionally disposed within one or more graphics processing unit (GPU).Rendering logic typically includes processing stages configured fordetermining the three-dimensional spatial relationships between objectsand/or for applying appropriate textures, etc., based on the game stateand viewpoint. The rendering logic can produce raw video that isencoded. For example, the raw video may be encoded according to an AdobeFlash® standard, HTML-5, .wav, H.264, H.263, On2, VP6, VC-1, WMA,Huffyuv, Lagarith, MPG-x. Xvid. FFmpeg, x264, VP6-8, realvideo, mp3, orthe like. The encoding process produces a video stream that isoptionally packaged for delivery to a decoder on a device. The videostream is characterized by a frame size and a frame rate. Typical framesizes include 800×600, 1280×720 (e.g., 720p), 1024×768, 1080p, althoughany other frame sizes may be used. The frame rate is the number of videoframes per second. A video stream may include different types of videoframes. For example, the H.264 standard includes a “P” frame and a “I”frame. I-frames include information to refresh all macro blocks/pixelson a display device, while P-frames include information to refresh asubset thereof. P-frames are typically smaller in data size than areI-frames. As used herein the term “frame size” is meant to refer to anumber of pixels within a frame. The term “frame data size” is used torefer to a number of bytes required to store the frame.

In some embodiments, the client can be a general purpose computer, aspecial purpose computer, a gaming console, a personal computer, alaptop computer, a tablet computer, a mobile computing device, aportable gaming device, a cellular phone, a set-top box, a streamingmedia interface/device, a smart television or networked display, or anyother computing device capable of being configured to fulfill thefunctionality of a client as defined herein. In one embodiment, a cloudgaming server is configured to detect the type of client device which isbeing utilized by the user, and provide a cloud-gaming experienceappropriate to the user's client device. For example, image settings,audio settings and other types of settings may be optimized for theuser's client device.

FIG. 10 illustrates an embodiment of an Information Service Providerarchitecture. Information Service Providers (ISP) 1070 delivers amultitude of information services to users 1082 geographically dispersedand connected via network 1086. An ISP can deliver just one type ofservice, such as stock price updates, or a variety of services such asbroadcast media, news, sports, gaming, etc. Additionally, the servicesoffered by each ISP are dynamic, that is, services can be added or takenaway at any point in time. Thus, the ISP providing a particular type ofservice to a particular individual can change over time. For example, auser may be served by an ISP in near proximity to the user while theuser is in her home town, and the user may be served by a different ISPwhen the user travels to a different city. The home-town ISP willtransfer the required information and data to the new ISP, such that theuser information “follows” the user to the new city making the datacloser to the user and easier to access. In another embodiment, amaster-server relationship may be established between a master ISP,which manages the information for the user, and a server ISP thatinterfaces directly with the user under control from the master ISP. Inanother embodiment, the data is transferred from one ISP to another ISPas the client moves around the world to make the ISP in better positionto service the user be the one that delivers these services.

ISP 1070 includes Application Service Provider (ASP) 1072, whichprovides computer-based services to customers over a network. Softwareoffered using an ASP model is also sometimes called on-demand softwareor software as a service (SaaS). A simple form of providing access to aparticular application program (such as customer relationshipmanagement) is by using a standard protocol such as HTTP. Theapplication software resides on the vendor's system and is accessed byusers through a web browser using HTML, by special purpose clientsoftware provided by the vendor, or other remote interface such as athin client.

Services delivered over a wide geographical area often use cloudcomputing. Cloud computing is a style of computing in which dynamicallyscalable and often virtualized resources are provided as a service overthe Internet. Users do not need to be an expert in the technologyinfrastructure in the “cloud” that supports them. Cloud computing can bedivided in different services, such as Infrastructure as a Service(IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).Cloud computing services often provide common business applicationsonline that are accessed from a web browser, while the software and dataare stored on the servers. The term cloud is used as a metaphor for theInternet (e.g., using servers, storage and logic), based on how theInternet is depicted in computer network diagrams and is an abstractionfor the complex infrastructure it conceals.

Further, ISP 1070 includes a Game Processing Server (GPS) 1074 which isused by game clients to play single and multiplayer video games. Mostvideo games played over the Internet operate via a connection to a gameserver. Typically, games use a dedicated server application thatcollects data from players and distributes it to other players. This ismore efficient and effective than a peer-to-peer arrangement, but itrequires a separate server to host the server application. In anotherembodiment, the GPS establishes communication between the players andtheir respective game-playing devices exchange information withoutrelying on the centralized GPS.

Dedicated GPSs are servers which run independently of the client. Suchservers are usually run on dedicated hardware located in data centers,providing more bandwidth and dedicated processing power. Dedicatedservers are the preferred method of hosting game servers for mostPC-based multiplayer games. Massively multiplayer online games run ondedicated servers usually hosted by the software company that owns thegame title, allowing them to control and update content.

Broadcast Processing Server (BPS) 1076 distributes audio or videosignals to an audience. Broadcasting to a very narrow range of audienceis sometimes called narrowcasting. The final leg of broadcastdistribution is how the signal gets to the listener or viewer, and itmay come over the air as with a radio station or TV station to anantenna and receiver, or may come through cable TV or cable radio (or“wireless cable”) via the station or directly from a network. TheInternet may also bring either radio or TV to the recipient, especiallywith multicasting allowing the signal and bandwidth to be shared.Historically, broadcasts have been delimited by a geographic region,such as national broadcasts or regional broadcast. However, with theproliferation of fast internet, broadcasts are not defined bygeographies as the content can reach almost any country in the world.

Storage Service Provider (SSP) 1078 provides computer storage space andrelated management services. SSPs also offer periodic backup andarchiving. By offering storage as a service, users can order morestorage as required. Another major advantage is that SSPs include backupservices and users will not lose all their data if their computers' harddrives fail. Further, a plurality of SSPs can have total or partialcopies of the user data, allowing users to access data in an efficientway independently of where the user is located or the device being usedto access the data. For example, a user can access personal files in thehome computer, as well as in a mobile phone while the user is on themove.

Communications Provider 1080 provides connectivity to the users. Onekind of Communications Provider is an Internet Service Provider (ISP)which offers access to the Internet. The ISP connects its customersusing a data transmission technology appropriate for delivering InternetProtocol datagrams, such as dial-up, DSL, cable modem, fiber, wirelessor dedicated high-speed interconnects. The Communications Provider canalso provide messaging services, such as e-mail, instant messaging, andSMS texting. Another type of Communications Provider is the NetworkService provider (NSP) which sells bandwidth or network access byproviding direct backbone access to the Internet. Network serviceproviders may consist of telecommunications companies, data carriers,wireless communications providers, Internet service providers, cabletelevision operators offering high-speed Internet access, etc.

Data Exchange 1088 interconnects the several modules inside ISP 1070 andconnects these modules to users 1082 via network 1086. Data Exchange1088 can cover a small area where all the modules of ISP 1070 are inclose proximity, or can cover a large geographic area when the differentmodules are geographically dispersed. For example, Data Exchange 1088can include a fast Gigabit Ethernet (or faster) within a cabinet of adata center, or an intercontinental virtual area network (VLAN).

Users 1082 access the remote services with client device 1084, whichincludes at least a CPU, a display and I/O. The client device can be aPC, a mobile phone, a netbook, tablet, gaming system, a PDA, etc. In oneembodiment, ISP 1070 recognizes the type of device used by the clientand adjusts the communication method employed. In other cases, clientdevices use a standard communications method, such as html, to accessISP 1070.

Embodiments of the present invention may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Theinvention can also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a wire-based or wireless network.

With the above embodiments in mind, it should be understood that theinvention can employ various computer-implemented operations involvingdata stored in computer systems. These operations are those requiringphysical manipulation of physical quantities. Any of the operationsdescribed herein that form part of the invention are useful machineoperations. The invention also relates to a device or an apparatus forperforming these operations. The apparatus can be specially constructedfor the required purpose, or the apparatus can be a general-purposecomputer selectively activated or configured by a computer programstored in the computer. In particular, various general-purpose machinescan be used with computer programs written in accordance with theteachings herein, or it may be more convenient to construct a morespecialized apparatus to perform the required operations.

The invention can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data, which can be thereafter be read by acomputer system. Examples of the computer readable medium include harddrives, network attached storage (NAS), read-only memory, random-accessmemory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical andnon-optical data storage devices. The computer readable medium caninclude computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the method operations were described in a specific order, itshould be understood that other operations may be performed in betweenoperations, or operations may be adjusted so that they occur at slightlydifferent times, or may be distributed in a system which allows theoccurrence of the processing operations at various intervals associatedwith the processing, as long as the processing of the overlay operationsare performed in the desired way.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the appended claims.

What is claimed is:
 1. A method, comprising, capturing video framesusing a camera, the video frames configured to capture markers on a headmounted display (HMD), the markers on the HMD are analyzed in thecaptured video frames to determine position and orientation of the HMD;and determining a separation distance between a first one of the markerson a front unit of the HMD and a second one of the markers on a rearsection of the HMD, the first one of the markers on the front unithaving first geometric configuration that is different than a secondgeometric configuration of the second one of the markers on the rearsection of the HMD, the first geometric configuration used to detect thefront unit of the HMD and the second geometric configuration used todetect the rear section of the HMD, the front unit of the HMD and therear section of the HMD being coupled together by an adjustable section,the determining of the separation distance includes analyzing theplurality of video frames captured by the camera to determine theseparation distance between the marker on the front unit and the markeron the rear section of the HMD; wherein the separation distance is setin response to placement of the HMD on a head of a user and a setting ofthe adjustable section, the separation distance being used duringfurther tracking of said position and orientation of the HMD, as themarkers on the front unit and the rear section are captured and analyzedfrom the captured video frames; wherein the determining the position andthe orientation of the HMD further includes processing inertial datafrom the HMD during processing of the plurality of the captured videoframes, the inertial data produces a set of values that quantify changesin rotation or changes in the position of the HMD; from time to time,continuing to determine the separation distance between the marker onthe front unit of the HMD and the marker on the rear section of the HMD,and determining if a new separation distance includes a difference fromthe separation distance, wherein the separation distance is currentlyused for the tracking; and using the new separation distance instead ofthe separation distance currently used when the difference exceeds apredefined allowable margin.
 2. The method of claim 1, wherein themarker on the rear section of the HMD includes a pair of markers, eachof the pair of markers on the rear section are provided with an LED forillumination, the pair of markers being defined on the rear section at arelative fixed separation.
 3. The method of claim 1, wherein the markeron the front unit and the marker on the rear section are identified inthe captured video frames when the user moves the HMD.
 4. The method ofclaim 3, further comprising, calibrating a geometric model of the HMD byupdating a current separation distance between the marker on the frontunit and the marker on the rear section with the separation distance. 5.The method of claim 4, wherein the separation distance is separatelyprocessed for a left side of the HMD and for a right side of the HMD,such each of the left side and the right side of the HMD includes amarker on the front unit of the HMD and a marker on the rear section ofthe HMD.
 6. The method of claim 1, wherein the front unit Includes afront face and side faces, and intersections between of the front faceand side faces define at least four transition corners, each of thetransition corners including a marker.
 7. The method of claim 6, whereinthe markers in the transition corners define a substantially rectangulararrangement when viewed from the camera.
 8. The method of claim 7,wherein each of the markers in the transition corners is defined by anlight emitting diodes (LED), the LEDs having an illumination shape thatpartially wraps the transition corners so that the illumination shape isdisposed partially on the front face and partially on the side faces. 9.The method of claim 1, wherein the markers are lights.
 10. A method,comprising, capturing video frames using a camera, the video framesconfigured to capture markers on a head mounted display (HMD), themarkers on the HMD are analyzed in the captured video frames todetermine position and orientation of the HMD; and determining aseparation distance between a first one of the markers on a front unitof the HMD and a second one of the markers on a rear section of the HMD,the first one of the markers on the front unit having first geometricconfiguration that is different than a second geometric configuration ofthe second one of the markers on the rear section of the HMD, the firstgeometric configuration used to detect the front unit of the HMD and thesecond geometric configuration used to detect the rear section of theHMD, the front unit of the HMD and the rear section of the HMD beingcoupled together by an adjustable section, the determining of theseparation distance includes analyzing the plurality of video framescaptured by the camera to determine the separation distance between themarker on the front unit and the marker on the rear section of the HMD;wherein the separation distance is set in response to placement of theHMD on a head of a user and a setting of the adjustable section, theseparation distance being used during further tracking of die positionand orientation of the HMD, as die markers on the front unit and therear section are captured and analyzed from the captured video frames;wherein determining the position and orientation of the HMD furtherincludes processing inertial data from the HMD during processing of theplurality of the captured video frames, the inertial data produces a setof values that quantify changes in rotation or changes in the positionof the HMD; wherein the rear section includes an adjuster for reducingor increasing size of the adjustable section of the HMD; and furthercomprising, receiving a signal indicative of a flag being set when anadjustment is made to the adjuster; and re-processing the separationdistance to produce a new separation distance in response to receivingthe signal.
 11. A head mounted display (HMD), comprising, a circuit forcommunicating with a computing system that processes multimedia contentfor display in the HMD; a front unit of the HMD having a screen fordisplaying multimedia content, the front unit having a first set ofLEDs; a rear section of the HMD having a second set of LEDs; and aconnector for adjustably coupling the front unit of the HMD with therear section of the HMD, such that adjustment of the connector changes aseparation distance between at least one of the first set of LEDs of thefront unit and at least one of the second set of LEDs of the rearsection; wherein the computing system processes image data captured ofthe HMD when worn by a user, the image data includes at least one of thefirst set of LEDs of the front unit and at least one of the second setof LEDs of the rear section to identify the separation distance betweenthe at least one of the first set of LEDs of the front unit and at leastone of the second set of LEDs of the rear section for a currentadjustment of the connector, wherein the set of LEDs of the front unitare arranged in a first pattern and the set of LEDs of the rear sectionare arranged in a second pattern, the first pattern being different thanthe second pattern to enable identification of the front unit ascompared to the rear section of the HMD, wherein the separation distanceis between at least one LED of the front unit and at least one LED ofthe rear section; an inertial sensor integrated in the HMD for trackingmovement of the HMD when worn by the user, the inertial sensor producingdata that is processed by the computing system along with the image datato track position and orientation of the HMD, wherein the separationdistance is used to calibrate data processed from the image data totrack the position and orientation; and further including an adjustmentdetector that produces a flag indicative of a change to the connector,the flag being processed by the computing system to triggerrecalibration of the separation distance.
 12. The HMD of claim 11,wherein the rear section includes an adjuster for adjusting a size ofthe connector, the size of the connector enabling expansion andcontraction for fitting of the HMD over a head of the user when the HMDis worn by the user.